from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	2.027753	0.260970	NaN	...	brute	-1	1	0.663	0.333881	0.012534	1.000	6.073273	6.077551	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.025990	0.002125	NaN	...	brute	-1	1	1.000	17.823298	0.265818	0.757	0.001458	0.001458	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.776739	0.044499	NaN	...	brute	-1	5	0.757	17.679337	0.066352	0.882	0.157061	0.157062	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.158312	0.024728	NaN	...	brute	1	100	0.882	0.334007	0.006963	1.000	6.461870	6.463274	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.023255	0.000925	NaN	...	brute	1	100	1.000	17.727084	0.051484	0.757	0.001312	0.001312	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.758314	0.045003	NaN	...	brute	-1	100	0.882	17.730265	0.042344	0.663	0.155571	0.155571	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.131869	0.024922	NaN	...	brute	1	5	0.757	0.265394	0.011031	1.000	8.032832	8.039768	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.022717	0.000942	NaN	...	brute	1	5	1.000	4.083309	0.043806	0.922	0.005563	0.005564	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.345079	0.016867	NaN	...	brute	1	1	0.663	4.171077	0.048267	0.929	0.322478	0.322499	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.637761	0.036030	NaN	...	brute	-1	1	0.896	0.265624	0.007487	1.000	6.165711	6.168160	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.005410	0.003306	NaN	...	brute	-1	1	1.000	4.144956	0.061061	0.922	0.001305	0.001305	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.464218	0.055673	NaN	...	brute	-1	5	0.922	4.122409	0.070032	0.896	0.597762	0.597848	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	6.665	0.0	-1	1	18.017	0.143	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.621	0.0	-1	5	0.324	0.008	1.000	0.037	0.037	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	6.541	0.0	1	100	17.684	0.045	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	6.265	0.0	-1	100	0.333	0.010	1.000	0.038	0.038	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.777	0.0	1	5	4.088	0.050	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	6.194	0.0	1	1	0.266	0.009	1.000	0.049	0.049	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.350	0.0	-1	1	4.162	0.071	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.348	0.0	-1	5	0.272	0.008	1.000	0.017	0.017	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.028	0.261	0.000	0.002	-1	1	0.334	0.013	1.000	6.073	6.078	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.002	0.000	0.026	-1	1	17.823	0.266	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.777	0.044	0.000	0.003	-1	5	17.679	0.066	0.882	0.157	0.157	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	0.000	0.025	-1	5	0.339	0.008	1.000	0.072	0.072	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.158	0.025	0.000	0.002	1	100	0.334	0.007	1.000	6.462	6.463	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	1	100	17.727	0.051	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.758	0.045	0.000	0.003	-1	100	17.730	0.042	0.663	0.156	0.156	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	0.000	0.024	-1	100	0.337	0.009	1.000	0.073	0.073	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.132	0.025	0.000	0.002	1	5	0.265	0.011	1.000	8.033	8.040	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	1	5	4.083	0.044	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.345	0.017	0.001	0.001	1	1	4.171	0.048	0.929	0.322	0.322	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.001	0.000	0.021	1	1	0.260	0.005	1.000	0.079	0.079	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.638	0.036	0.000	0.002	-1	1	0.266	0.007	1.000	6.166	6.168	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.003	0.000	0.005	-1	1	4.145	0.061	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.464	0.056	0.000	0.002	-1	5	4.122	0.070	0.896	0.598	0.598	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.002	0.000	0.007	-1	5	0.262	0.006	1.000	0.026	0.026	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.882532	1.124641	NaN	...	kd_tree	-1	1	0.929	2.949033	0.275870	1.000	0.299261	0.300568	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.003396	0.000762	NaN	...	kd_tree	-1	1	1.000	138.313347	0.000000	0.946	0.000025	0.000025	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.098293	0.383205	NaN	...	kd_tree	-1	5	0.946	140.391716	0.000000	0.951	0.007823	0.007823	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.737746	0.794953	NaN	...	kd_tree	1	100	0.951	2.944724	0.187431	1.000	1.948483	1.952426	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003289	0.000942	NaN	...	kd_tree	1	100	1.000	141.171351	0.000000	0.946	0.000023	0.000023	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.216986	0.250691	NaN	...	kd_tree	-1	100	0.951	139.253027	0.000000	0.929	0.023102	0.023102	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.659354	0.213759	NaN	...	kd_tree	1	5	0.946	0.006020	0.000230	1.000	275.634362	275.835459	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001505	0.000416	NaN	...	kd_tree	1	5	1.000	0.047670	0.006143	0.911	0.031572	0.031833	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.936813	0.277644	NaN	...	kd_tree	1	1	0.929	0.066047	0.002241	0.894	14.184025	14.192185	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.035003	0.014955	NaN	...	kd_tree	-1	1	0.891	0.005915	0.000503	1.000	5.918086	5.939464	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.002949	0.000312	NaN	...	kd_tree	-1	1	1.000	0.043179	0.002485	0.911	0.068293	0.068406	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.031504	0.004727	NaN	...	kd_tree	-1	5	0.911	0.043827	0.002733	0.891	0.718830	0.720227	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.844	0.046	0.028	0.0	-1	1	140.461	0.000	0.929	0.020	0.020	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.413	0.068	0.018	0.0	-1	5	2.966	0.331	1.000	1.488	1.497	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.395	0.042	0.018	0.0	1	100	140.524	0.000	0.951	0.031	0.031	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.387	0.056	0.018	0.0	-1	100	2.982	0.267	1.000	1.471	1.477	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.472	0.085	0.018	0.0	1	5	0.047	0.016	0.891	95.243	100.830	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.419	0.056	0.018	0.0	1	1	0.006	0.000	1.000	728.000	728.843	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.021	0.0	-1	1	0.066	0.003	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.015	0.0	-1	5	0.006	0.001	1.000	0.182	0.183	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.883	1.125	0.000	0.001	-1	1	2.949	0.276	1.000	0.299	0.301	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	138.313	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.098	0.383	0.000	0.001	-1	5	140.392	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.002	0.000	0.004	-1	5	2.980	0.326	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.738	0.795	0.000	0.006	1	100	2.945	0.187	1.000	1.948	1.952	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	141.171	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.217	0.251	0.000	0.003	-1	100	139.253	0.000	0.929	0.023	0.023	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.006	0.001	0.000	0.006	-1	100	3.009	0.318	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.659	0.214	0.000	0.002	1	5	0.006	0.000	1.000	275.634	275.835	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.048	0.006	0.911	0.032	0.032	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.937	0.278	0.000	0.001	1	1	0.066	0.002	0.894	14.184	14.192	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.203	0.204	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.035	0.015	0.000	0.000	-1	1	0.006	0.001	1.000	5.918	5.939	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	1	0.043	0.002	0.911	0.068	0.068	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.032	0.005	0.001	0.000	-1	5	0.044	0.003	0.891	0.719	0.720	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.006	0.001	1.000	0.405	0.417	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.165556	0.008624	300	...	100	100	10	0.824	0.444116	0.01947	1.0	0.372776	0.373134	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	125.267	0.0	300	0.001	0.001	0.534	0.026	0.824	234.499	234.786	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.166	0.009	300	0.005	0.0	0.444	0.019	1.0	0.373	0.373	See	See